While there has been significant-work in face detection (see, for example, Nguyen, D., Halupka, D., Aarabi, P., Sheikholesiami, A., “Real-time Face Localization Using field Programmable Gate Arrays”, IEEE Transactions on Systems, Man, and Cybernetics, Part B. Vol 36, No. 4, pp. 902-912, August 2006), there seems to have been little work in the area of body analysis for the purpose of recommending, products or treatments and means of visually displaying possible outcomes of applications of products or treatments.
Specifically, U.S. Pat. No. 6,293,284 granted to Rigg describes a method and apparatus utilizing manual user interaction in order to recolor the facial features and to simulate the effects of cosmetic products. Unfortunately, this approach does not utilize advanced image processing, computer vision or machine learning methodologies and does not simulate applications of products or treatments. A user has to spend significant time and effort in order to manually enter the parameters for the facial recoloring.
Virtual, plastic surgery is the focus of U.S. Pat. Nos. 5,854,850 and 5,825,941 to granted Linford et al. and U.S. Pat. No. 5,687,259 to granted Linford. However, the system disclosed in these references is relatively complicated, and is intended to be an in-clinic system used by professional or experienced operators. Furthermore, the system is not provided on the Internet or through mobile and wireless devices, and does not address utilization of advanced image processing, computer vision or machine learning methodologies for estimating the plastic surgery parameters. As a result, operators are required to manually adjust the system parameters in order to display the results of plastic surgery in a virtual fashion. This system is mostly manual, and does not utilize face localization, feature detection, feature/face recoloring, or visual display of applications of products or treatments on an automatic or semi-automatic basis.
The method disclosed in U.S. Pat. No. 6,502,583 granted to Utsugi utilizes image processing in order to simulate the effects of makeup on a target face. This system, however, does not utilize automatic or semi-automatic face detection, feature detection, or parameter estimation and as a result requires manual user input for estimating the necessary parameters. Furthermore, this system was not intended for general virtual face modifications based on applications of products or treatments.
The system and method of U.S. Pat. No. 6,937,755 granted to Orpaz discloses a manual method for visually demonstrating make-up cosmetics and fashion accessories. This visualization requires manual user inputs in order to work effectively (i.e. it is neither automatic nor semi-automatic), and does not allow for advanced face modifications, or face feature e-coloring and replacement to visualize applications of products or treatments on an automatic or semi-automatic basis.
A system and method is disclosed in U.S. Pat. No. 5,495,338 granted to Gouriou et al. which utilizes eye information (such as the inner eye colors) in order to estimate the ideal eye makeup for a given eye. However, this approach is purely a cosmetics suggestion system; it does not perform any face adjustment, or face recoloring automatically, semi-automatically, or even manually.
U.S. Pat. No. 5,659,625 granted to Marquardt discloses a method involving a geometric model to fit the face. These geometric models can be used tor face animation as well as for cosmetics applications. However, this system does not achieve automatic or semi-automatic feature modification based on applications of products or treatments.
A method tor locating the lips of a face by bandpass filtering is described in U.S. Pat. No. 5,805,745 granted to Graf. However, this reference does not disclose a means for defecting other features of the face, nor does it describe automatic or semi-automatic face modifications. Furthermore, the bandpass filtering method is unsophisticated, and does not involve feature extraction methods utilizing edge, color and/or shape information, or relative feature and face information processing in order to accurately locate the facial features.
The method and apparatus described in U.S. Pat. No. 5,933,527 granted to Ishikawa allows a user to specify a search range that is then used to search for specific facial features. However, the approach taught therein is not capable of automatic facial feature detection, and is incapable of automatic or semi-automatic advancement face processing algorithms.
The invention disclosed in U.S. Pat. No. 7,079,158 granted to Lambertsen is a virtual makeover system and method. However, the reference does not disclose a means for virtual, operations on the face or automatic or semi-automatic advanced face modification, and suffers from a relatively complicated user interface.
The invention disclosed in U.S. patent application Ser. No. 13/813,003 (Publication. No. 20130129210) invented by Seung Won Na discloses a recommendation system based on the recognition of a face and style. This invention is not operable to process multiple images or video, or to preview a recommended look on a user's live video or image. Furthermore, it does not comprise a voting or neural network system for combining results of multiple images.
The invention disclosed in U.S. patent application Ser. No. 13/400,980 (Publication No. 20120223956) invented by Mari Saito and Tatsuki Kashitani is a method for recognizing the facial image in image data, and recognizing facial features of the facial image. A makeup image providing information assisting in the application of makeup is generated and it is superimposed on the facial image to produce output image data. This invention is not operable as a product or treatment recommendation system or method that utilizes multiple images or videos, or a means to preview a recommended product or treatment on a user's live video image. Furthermore, it does not comprise a voting or neural network system or method.
The invention disclosed in U.S. patent application Ser. No. 11/233,599 (Publication No. 20070058858) invented by Michael Harville et al., does not comprise a product recommendation system or method operable based on multiple images or videos. Furthermore, it does not disclose a voting or neural network system or method, or a means for previewing a recommended product or treatment on a user's live video image.
The invention disclosed in U.S. Pat. No. 8,666,844 granted to Johnson & Johnson Consumer Companies discloses a method for predicting or recommending commercially available products based upon input received from consumers. This patent does not disclose the use of any images or videos to detect features of a user in the process of recommending products to a user.
The invention disclosed in U.S. Pat. No. 7,376,346 granted to J&J Consumer Companies, Inc. discloses a method tor assessing the efficacy of skin care products by comparing images of the skin prior to and after use of said skin care product. This patent does not disclose the use of any images or videos to detect features of a user in the process of assessing the efficacy of a product.
The invention disclosed in U.S. Pat. No, 8,661,050 granted to Roy Varshavsky, Moshe Tennenholtz, and Ron Karidi, discloses a system that utilizes relationships between objects and search results for making recommendations. This patent does not disclose the use of any images or videos to detect features of a user in the process of recommending products to a user.
In addition to these prior art references, there are several systems provided on the Internet that are operable to perform manual face modification, for example, EZface™, Approach Infinity Media™. However, none of these systems are capable of face feature modification, advanced face processing, either automatic or semi-automatic. As well, all of these systems employ Macromedia™ flash technology which places a significant computational burden on the client/user computers and is not easily capable of being widely employed on mobile phones and handheld computers. Finally, the user interface complexity of all these systems is problematic as they are generally difficult to use, complicated to adjust, and far more elaborate to use than a simple “choose and modify” approach.
U.S. Pat. No. 8,660,319 granted on Feb. 25, 2014, U.S. Pat. No. 8,620,038 granted Dec. 31, 2013, U.S. Pat. No. 6,265,351 granted Sep. 11, 2012, and U.S. Pat. No. 8,611,673 granted Dec. 17, 2103 to Parham Aarabi, as well as U.S. Patent Application Publication No. 2015/0120719 invented by Parham Aarabi and published on Apr. 30, 2015, and U.S. patent application Ser. No. 14/185,321 invented by Parham Aarabi and filed on Feb. 20, 2014, disclose inventions relating to detecting elements in images. However, none of these prior art references discloses the use of any images or videos to detect features of a user in the process of recommending products to a user.
In view, of the foregoing, what are needed are methods and systems tor modifying digital face images that overcome the limitations of the prior art described above. In particular, what is needed is a method and system employing advanced, detection and localization techniques for enabling recommendations of products and treatments and automatic and/or semi-automatic image modification to display applications of products or treatments. Furthermore, what is needed is a method, and system where such recommendations and facial modifications are processed on host servers instead of the user computers. In addition, what is needed is a method and system that is simple, easy to use, and capable of being implemented on a variety of devices.